<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Transform limitations | ElasticSearch 7.7 权威指南中文版</title>
	<meta name="keywords" content="ElasticSearch 权威指南中文版, elasticsearch 7, es7, 实时数据分析，实时数据检索" />
    <meta name="description" content="ElasticSearch 权威指南中文版, elasticsearch 7, es7, 实时数据分析，实时数据检索" />
    <!-- Give IE8 a fighting chance -->
    <!--[if lt IE 9]>
    <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
    <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->
	<link rel="stylesheet" type="text/css" href="../static/styles.css" />
	<script>
	var _link = 'transform-limitations.html';
    </script>
</head>
<body>
<div class="main-container">
    <section id="content">
        <div class="content-wrapper">
            <section id="guide" lang="zh_cn">
                <div class="container">
                    <div class="row">
                        <div class="col-xs-12 col-sm-8 col-md-8 guide-section">
                            <div style="color:gray; word-break: break-all; font-size:12px;">原英文版地址: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/transform-limitations.html" rel="nofollow" target="_blank">https://www.elastic.co/guide/en/elasticsearch/reference/7.7/transform-limitations.html</a>, 原文档版权归 www.elastic.co 所有<br/>本地英文版地址: <a href="../en/transform-limitations.html" rel="nofollow" target="_blank">../en/transform-limitations.html</a></div>
                        <!-- start body -->
                  <div class="page_header">
<strong>重要</strong>: 此版本不会发布额外的bug修复或文档更新。最新信息请参考 <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html" rel="nofollow">当前版本文档</a>。
</div>
<div id="content">
<div class="breadcrumbs">
<span class="breadcrumb-link"><a href="index.html">Elasticsearch Guide [7.7]</a></span>
»
<span class="breadcrumb-link"><a href="data-rollup-transform.html">Roll up or transform your data</a></span>
»
<span class="breadcrumb-link"><a href="transforms.html">Transforming data</a></span>
»
<span class="breadcrumb-node">Transform limitations</span>
</div>
<div class="navheader">
<span class="prev">
<a href="transform-troubleshooting.html">« Troubleshooting transforms</a>
</span>
<span class="next">
<a href="high-availability.html">Set up a cluster for high availability »</a>
</span>
</div>
<div class="section xpack">
<div class="titlepage"><div><div>
<h2 class="title">
<a id="transform-limitations"></a>Transform limitations<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a><a class="xpack_tag" href="https://www.elastic.co/subscriptions"></a>
</h2>
</div></div></div>

<p>The following limitations and known problems apply to the 7.7.1 release of
the Elastic transform feature:</p>
<h4>
<a id="transform-ui-limitation"></a>Transforms UI will not work during a rolling upgrade from 7.2<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>If your cluster contains mixed version nodes, for example during a rolling
upgrade from 7.2 to a newer version, and transforms have been created in 7.2,
the transforms UI (earler data frame UI) will not work. Please wait until all
nodes have been upgraded to the newer version before using the transforms UI.</p>
<h4>
<a id="transform-rolling-upgrade-limitation"></a>Transforms reassignment suspended during a rolling upgrade from 7.2 and 7.3<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>If your cluster contains mixed version nodes, for example during a rolling
upgrade from 7.2 or 7.3 to a newer version, transforms whose nodes are stopped will
not be reassigned until the upgrade is complete. After the upgrade is done, transforms
resume automatically; no action is required.</p>
<h4>
<a id="transform-datatype-limitations"></a>Data frame data type limitation<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>Data frames do not (yet) support fields containing arrays – in the UI or
the API. If you try to create one, the UI will fail to show the source index
table.</p>
<h4>
<a id="transform-kibana-limitations"></a>Up to 1,000 transforms are supported<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>A single cluster will support up to 1,000 transforms. When using the
<a href="get-transform.html" class="ulink" target="_top">GET transforms API</a> a total <code class="literal">count</code> of transforms
is returned. Use the <code class="literal">size</code> and <code class="literal">from</code> parameters to enumerate through the full
list.</p>
<h4>
<a id="transform-aggresponse-limitations"></a>Aggregation responses may be incompatible with destination index mappings<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>When a transform is first started, it will deduce the mappings
required for the destination index. This process is based on the field types of
the source index and the aggregations used. If the fields are derived from
<a href="search-aggregations-metrics-scripted-metric-aggregation.html" class="ulink" target="_top"><code class="literal">scripted_metrics</code></a>
or <a href="search-aggregations-pipeline-bucket-script-aggregation.html" class="ulink" target="_top"><code class="literal">bucket_scripts</code></a>,
<a href="dynamic-mapping.html" class="ulink" target="_top">dynamic mappings</a> will be used. In some instances the
deduced mappings may be incompatible with the actual data. For example, numeric
overflows might occur or dynamically mapped fields might contain both numbers
and strings. Please check Elasticsearch logs if you think this may have occurred. As a
workaround, you may define custom mappings prior to starting the
transform. For example,
<a href="indices-create-index.html" class="ulink" target="_top">create a custom destination index</a> or
<a href="indices-templates.html" class="ulink" target="_top">define an index template</a>.</p>
<h4>
<a id="transform-batch-limitations"></a>Batch transforms may not account for changed documents<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>A batch transform uses a
<a href="search-aggregations-bucket-composite-aggregation.html" class="ulink" target="_top">composite aggregation</a>
which allows efficient pagination through all buckets. Composite aggregations
do not yet support a search context, therefore if the source data is changed
(deleted, updated, added) while the batch data frame is in progress, then the
results may not include these changes.</p>
<h4>
<a id="transform-consistency-limitations"></a>Continuous transform consistency does not account for deleted or updated documents<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>While the process for transforms allows the continual recalculation of the
transform as new data is being ingested, it does also have some limitations.</p>
<p>Changed entities will only be identified if their time field has also been
updated and falls within the range of the action to check for changes. This has
been designed in principle for, and is suited to, the use case where new data is
given a timestamp for the time of ingest.</p>
<p>If the indices that fall within the scope of the source index pattern are
removed, for example when deleting historical time-based indices, then the
composite aggregation performed in consecutive checkpoint processing will search
over different source data, and entities that only existed in the deleted index
will not be removed from the data frame destination index.</p>
<p>Depending on your use case, you may wish to recreate the transform entirely
after deletions. Alternatively, if your use case is tolerant to historical
archiving, you may wish to include a max ingest timestamp in your aggregation.
This will allow you to exclude results that have not been recently updated when
viewing the destination index.</p>
<h4>
<a id="transform-deletion-limitations"></a>Deleting a transform does not delete the destination index or Kibana index pattern<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>When deleting a transform using <code class="literal">DELETE _transform/index</code>
neither the destination index nor the Kibana index pattern, should one have been
created, are deleted. These objects must be deleted separately.</p>
<h4>
<a id="transform-aggregation-page-limitations"></a>Handling dynamic adjustment of aggregation page size<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>During the development of transforms, control was favoured over performance.
In the design considerations, it is preferred for the transform to take longer
to complete quietly in the background rather than to finish quickly and take
precedence in resource consumption.</p>
<p>Composite aggregations are well suited for high cardinality data enabling
pagination through results. If a <a href="circuit-breaker.html" class="ulink" target="_top">circuit breaker</a>
memory exception occurs when performing the composite aggregated search then we
try again reducing the number of buckets requested. This circuit breaker is
calculated based upon all activity within the cluster, not just activity from
transforms, so it therefore may only be a temporary resource
availability issue.</p>
<p>For a batch transform, the number of buckets requested is only ever adjusted
downwards. The lowering of value may result in a longer duration for the
transform checkpoint to complete. For continuous transforms, the number of buckets
requested is reset back to its default at the start of every checkpoint and it
is possible for circuit breaker exceptions to occur repeatedly in the Elasticsearch logs.</p>
<p>The transform retrieves data in batches which means it calculates several
buckets at once. Per default this is 500 buckets per search/index operation. The
default can be changed using <code class="literal">max_page_search_size</code> and the minimum value is 10.
If failures still occur once the number of buckets requested has been reduced to
its minimum, then the transform will be set to a failed state.</p>
<h4>
<a id="transform-dynamic-adjustments-limitations"></a>Handling dynamic adjustments for many terms<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>For each checkpoint, entities are identified that have changed since the last
time the check was performed. This list of changed entities is supplied as a
<a href="query-dsl-terms-query.html" class="ulink" target="_top">terms query</a> to the transform composite
aggregation, one page at a time. Then updates are applied to the destination
index for each page of entities.</p>
<p>The page <code class="literal">size</code> is defined by <code class="literal">max_page_search_size</code> which is also used to
define the number of buckets returned by the composite aggregation search. The
default value is 500, the minimum is 10.</p>
<p>The index setting
<a href="index-modules.html#dynamic-index-settings" class="ulink" target="_top"><code class="literal">index.max_terms_count</code></a> defines
the maximum number of terms that can be used in a terms query. The default value
is 65536. If <code class="literal">max_page_search_size</code> exceeds <code class="literal">index.max_terms_count</code> the
transform will fail.</p>
<p>Using smaller values for <code class="literal">max_page_search_size</code> may result in a longer duration
for the transform checkpoint to complete.</p>
<h4>
<a id="transform-scheduling-limitations"></a>Continuous transform scheduling limitations<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>A continuous transform periodically checks for changes to source data. The functionality
of the scheduler is currently limited to a basic periodic timer which can be
within the <code class="literal">frequency</code> range from 1s to 1h. The default is 1m. This is designed
to run little and often. When choosing a <code class="literal">frequency</code> for this timer consider
your ingest rate along with the impact that the transform
search/index operations has other users in your cluster. Also note that retries
occur at <code class="literal">frequency</code> interval.</p>
<h4>
<a id="transform-failed-limitations"></a>Handling of failed transforms<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>Failed transforms remain as a persistent task and should be handled
appropriately, either by deleting it or by resolving the root cause of the
failure and re-starting.</p>
<p>When using the API to delete a failed transform, first stop it using
<code class="literal">_stop?force=true</code>, then delete it.</p>
<h4>
<a id="transform-availability-limitations"></a>Continuous transforms may give incorrect results if documents are not yet available to search<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h4>
<p>After a document is indexed, there is a very small delay until it is available
to search.</p>
<p>A continuous transform periodically checks for changed entities between the time since
it last checked and <code class="literal">now</code> minus <code class="literal">sync.time.delay</code>. This time window moves
without overlapping. If the timestamp of a recently indexed document falls
within this time window but this document is not yet available to search then
this entity will not be updated.</p>
<p>If using a <code class="literal">sync.time.field</code> that represents the data ingest time and using a
zero second or very small <code class="literal">sync.time.delay</code>, then it is more likely that this
issue will occur.</p>
<div class="section">
<div class="titlepage"><div><div>
<h3 class="title">
<a id="transform-date-nanos"></a>Support for date nanoseconds data type<a class="edit_me edit_me_private" rel="nofollow" title="Editing on GitHub is available to Elastic" href="https://github.com/elastic/elasticsearch/edit/7.7/docs/reference/transform/limitations.asciidoc">edit</a>
</h3>
</div></div></div>
<p>If your data uses the <a class="xref" href="date_nanos.html" title="Date nanoseconds datatype">date nanosecond data type</a>, aggregations
are nonetheless on millisecond resolution. This limitation also affects the
aggregations in your transforms.</p>
</div>

</div>
<div class="navfooter">
<span class="prev">
<a href="transform-troubleshooting.html">« Troubleshooting transforms</a>
</span>
<span class="next">
<a href="high-availability.html">Set up a cluster for high availability »</a>
</span>
</div>
</div>

                  <!-- end body -->
                        </div>
                        <div class="col-xs-12 col-sm-4 col-md-4" id="right_col">
                        
                        </div>
                    </div>
                </div>
            </section>
        </div>
    </section>
</div>
<script src="../static/cn.js"></script>
</body>
</html>